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The  September  2008  issue  of  Information  Systems  Test  fsf  Evaluation  focused  on  improving 
suitability.  It  coincided  with,  and  in  some  cases  led,  reliability  improvement  policy  and 
guidance.  By  early  September  2008,  all  the  military  departments  responded  positively  to  the 
direction  of  Under  Secretary  Young  in  July  2008  to  “establish  a  reliability  improvement 
acquisition  policy  to  address  the  problem  of  inadequate  system  RAM  [reliability- 
availability-maintainability].  ”  For  example,  on  September  4,  2008,  Dr.  Donald  Winter, 
Secretary  of  the  Navy,  stated,  “Having  performance  is  important,  but  not  as  important  in 
most  cases,  as  having  reliability.  ”  I  offer  several  specific  examples  of  progress  made  toward 
improving  suitability  and  reliability  and  a  guide  on  where  to  find  the  products. 


n  June  6,  2008,  the  Defense  Science 
Board  Task  Force  on  Developmental 
Test  and  Evaluation  (which  focused 
on  improving  reliability)  released 
its  final  report.  It  is 
available  at  http;//www.acq.osd.mil/ 
sse/ dte/  docs/DSB  -Rpt-DTE-May2008 . 
pdf.  Charged  to  implement  key  Defense 
Science  Board  recommendations,  the 
U.S.  Department  of  Defense  (DoD) 

Reliability  Improvement  Work  Group 
(RIWG)  worked  from  March  to  August 
2008  to: 

•  ensure  programs  are  formulated 
to  execute  a  viable  systems  engi¬ 
neering  strategy  from  the  begin¬ 
ning,  including  a  reliability- avail¬ 
ability-maintainability  (RAM) 
growth  program,  as  an  integral 
part  of  design  and  development  (that  is.  Start 
Programs  Right)', 

•  ensure  government  organizations  reconstitute  a 
cadre  of  experienced  Test  and  Evaluation  (T&E) 
and  RAM  personnel  {Re-enforce  the  Work  Force)', 

•  implement  mandated  integrated  developmental 
test  and  operational  test,  including  the  sharing 
and  access  to  all  appropriate  contractor  and 
government  data  and  the  use  of  operationally 
representative  environments  in  early  testing 
{Implement  Integrated  Testing). 

On  September  4  the  RIWG  published  its  final 
report  containing  implementing  actions  and  products. 


along  with  military  departments’  implementing  steps 
for  those  actions  and  products.  The  report  is  available 
at  http:/ /www.acq.osd.mil/  sse/ dte/ spec-studies.html. 
RIWG  products  were  developed  by  representatives 
from  all  military  departments  and 
the  Defense  Information  Systems 
Agency  (DISA).  It  is  widely  agreed 
that  these  products  are  comprehen¬ 
sive  and,  if  implemented,  wiU  be 
effective  in  achieving  proper  defense 
system  reliability  and  thereby  con¬ 
taining  system  sustainment  costs.  A 
short  summary  of  RIWG  products 
includes  the  following. 

Start  programs  right 

Department  policy  for 
reliability  improvement 

Issued  on  July  21,  2008,  it  states: 
"It  is  Department  policy  for  programs  to  be 
formulated  to  execute  a  viable  RAM  strategy  that 
includes  a  reliability  growth  program  as  an 
integral  part  of  design  and  development.  Addi¬ 
tionally,  RAM  shall  be  integrated  within  the 
systems  engineering  processes,  documented  in  the 
program’s  Systems  Engineering  Plan  and  Life 
Cycle  Sustainment  Plan,  and  assessed  during 
technical  reviews,  test  and  evaluation,  and 
program  support  reviews.  This  policy  will  be 
included  in  the  DoD  Instruction  5000.2.” 

The  secretaries  of  the  military  departments  were 
directed  to  establish  their  own  reliability  improvement 
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acquisition  policy.  The  policy  memo  is  at  http://www. 
acq.osd.mil/sse/dte/docs/USD-ATLMemo-RAM-Policy- 
21Jul08.pdf. 

Sample  reliability  language  for  Sections  C,  L, 
and  M  of  acquisition  contracts 

It  also  includes  a  checklist  for  evaluating  reliahility 
program  plans  and  a  sample  performance  incentive  fee. 
For  example,  from  the  sample  language  for  Section  C 
(Statement  of  Work): 

“The  contractor  shall  develop  a  reliability  model 
for  the  system.  At  a  minimum,  the  system 
reliability  model  shall  be  used  to:  (1)  generate 
and  update  the  reliability  allocations  from  the 
system  level  down  to  lower  indenture  levels;  (2) 
aggregate  system-level  reliability  based  on  reli¬ 
ability  estimates  from  lower  indenture  levels;  (3) 
identify  single  points  of  failure;  and  (4)  identify 
reliability-critical  items  and  areas  where  addi¬ 
tional  design  or  testing  activities  are  required  in 
order  to  achieve  the  reliability  requirements.  ” 

The  contract  language  is  at  https://acc.dau.mil/ 
CommunityBrowser.aspxPid =219127 8dang = en-US . 

The  language  and  practices  are  consistent  with  a  new 
industry  standard  for  reliability:  GEIA- STD-0009: 
Reliability  Program  Standard  for  Systems  Design, 
Development,  and  Manufacturing,  August  2008. 

Program  reliability  and  maintainability 
review  template 

This  tool  defines  relevant  reliability  activities  and 
evidence  of  them  associated  with  16  technical  reviews 
across  system  acquisition.  For  example,  in  the  initial 
technical  review  during  concept  refinement,  look  for 
evidence  of  documented  reliability  assumptions  and 
supporting  rationale  accompanying  reliability  require¬ 
ments  for  the  preferred  solution.  The  template  is 
available  at  http://www.acq.osd.mil/sse/docs/RAM- 
Planning-T  emplate.xls . 

Standard  evaluation  criteria 

To  determine  if  system  contractors  employ  practices 
needed  to  achieve  reliability  requirements,  a  scorecard 
offers  criteria  and  a  scoring  means.  For  example,  one 
criterion  is  “sufficiently-sized  reliability  engineering 
staff  directly  tied  to  design  team.”  The  contractor  score 
is  determined  by  the  size  of  the  reliability  engineering 
staff,  its  workload,  and  its  communication  with  the 
system  design  team.  The  scorecard  is  at  https://acc.dau. 
miFCommunityBrowser.aspx?id=210483Sdang=en-US. 

RAM  champions  in  each  Service 

The  RIWG  recommended  designated  Service 
“champions”  to  ensure  reliability  initiatives  become 


institutionalized.  For  example,  the  U.S.  Army  has 
named  a  headquarters  executive  to  serve  as  the 
Department  of  the  Army  Reliability  Chief. 

Re-enforce  the  work  force 

Strengthen  Defense  Acquisition 
University  curriculum 

RIWG  representatives  recommended  Defense  Ac¬ 
quisition  University  curricula  and  work  force  certifica¬ 
tion  changes  to  the  Overarching-Functional  Integrated 
Product  Team.  There  was  broad  concurrence  that  the 
Defense  Acquisition  University  can  further  the  intent 
for  RAM  effectiveness  by  coordinating  the  education  of 
the  functional  work  forces  that  play  pivotal  roles  at 
various  life  cycle  stages  of  systems  development.  Most 
recently,  the  RIWG  proposed  rehabUity  contracting 
subject  matter  for  contracting  work  force  education. 

Implement  integrated  testing 

Integrated  testing  defined 

A  memorandum  formally  defines  integrated  T&E  as 
a  basis  for  developing  further  guidance.  The  memo¬ 
randum  is  at  http://www.acq.osd.mil/sse/dte/docs/ 
SecDefMemo-Definition-of-Integrated-T  esting-25  Apr08 . 
pdf 

Early  T&E  involvement  in  RFPs  [Request  for 
Proposals]  and  contracts.  The  RIWG  included  data 
sharing  concepts  in  a  guidebook  titled  “Incorporating 
Test  and  Evaluation  into  Department  of  Defense 
Acquisition  Contracts.”  The  guidebook  is  located  at 
http://www.acq.osd.mil/sse/dte/guidance.html. 

New  Defense  Acquisition  Guidebook  Guidance  and  T&E 
Master  Plan  format.  New  guidance  in  the  Defense 
Acquisition  Guidebook  Guidance  and  a  newTScE  Master 
Plan  format  emphasize  integrated  T6cE.  It  is  available  at 
https://  akss.dau.miL/DAG/ Guidebook/I  G_c9.0.asp. 

Service  implementation  responses 

Across  the  DoD,  implementation  of  the  actions  in 
the  RIWG  report  is  in  progress.  The  Army  has  been 
aggressive  in  emphasizing  reliability  accountability, 
contracting  for  reliability,  and  establishing  an  early 
warning  mechanism  for  potential  reliability  problems. 
The  U.S.  Navy  is  reinvigorating  their  RAM  processes 
and  implementing  reforms  such  as  use  of  the  new 
GEIA  standard  in  developing  program  requirements 
(that  standard,  GEIA- STD-0009  Reliability  Program 
Standard  for  System  Design,  Development  and 
Manufacturing,  is  available  through  the  IT/\A  Gov¬ 
ernment  Electronics  and  Information  Technology 
Association  On-Line  Web  Store  at  http://www. 
techstreet.com/cgi-bin/detailPproduct_id  =  1568406). 
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Initiatives  to  Improve  Reliability, 
Maintainability,  and  Availability 

Reliability,  Availability,  Maintainability  and  Cost  Manual  (RAM-C  Manual) 
DOT&E  on  JCIDS  Functional  Control  Boards 

GEIA  Standard  009,  RFP  and  Contract  Language,  Investment  Model 

Reliability  Growth  in  design  phase 


RAM  growth  monitoring  for  incentives, 
Young/Bolton  memo 


RAM  program  Evaluation  and 
Standards,  testing  KPP 

RAM  field  data 
collection,  feedback 
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Figure  1.  Result  of  reliability  improvement — fewer  faiiures,  reduced  ownership  cost 


The  U.S.  Air  Force  is  examining  their  policies  and 
processes  to  improve  RAM. 

Department  leadership  reviewed  RIWG  actions  and 
products  and  Services  implementation  in  late  Septem¬ 
ber  and  directed  a  follow-up  review  in  December  2008 
to  assess  the  extent  of  implementation. 

With  this  emphasis  across  the  department,  it  is 
reasonable  to  expect  to  see  improved  reliability  in 
programs.  We  will  not  see  results  overnight,  as  it  must 
begin  with  requirements  and  contracts  or  those 
opportunities  that  occur  in  any  program  restructuring. 
The  end  result  of  these  initiatives  will  be  reduced 
system  ownership  costs,  fewer  failure  modes,  improved 
reliability,  and  improved  system  value  to  our  warfight¬ 
ers,  as  Figure  1  illustrates. 

Of  course,  testers  have  a  vital  role.  RAM  expertise  is 
in  short  supply  across  the  DoD  (and  industry).  TScE 


can  step  forward,  join  requirements  efforts,  influence 
program  office  System  Engineering  Plans  and  T6cE 
Master  Plans  to  properly  plan  effective  reliability 
programs,  and  measure  and  report  results.  Only  with 
T&E  data  will  programs  know  where  they  stand  with 
respect  to  failure  modes,  demonstrated  system  reliabil¬ 
ity,  and  the  reliability  growth  potential.  □ 

Dr.  Ernest  Seglie  is  science  advisor,  Office  of  the 
Director,  Operational  Test  and  Evaluation  (DOT^E), 
the  Pentagon,  Washington,  D.C.  He  provides  scientific 
and  technical  guidance  on  the  overall  approach  to 
Department  of  Defense  (DoD)  evaluation  of  the  opera¬ 
tional  effectiveness  and  suitability  of  major  DoD  weapon 
systems,  provides  technical  review  of  test  reports,  and  serves 
as  chief  technical  advisor  to  the  Director,  DOT^E. 
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